How To Use An Amazon S3 Connection Profile for Google Cloud Access
Earlier in the year, Google made an announcement that they would discontinue support for JSON-RPC and global HTTP batch endpoints. This has a huge impact on Google Cloud connections in ChronoSync and InterConneX. You can no longer set up and use a Google Cloud Storage connection to access your Google Cloud Storage. But, fear not! This guide shows you how you can now use an Amazon S3 connection profile to connect and sync/backup to your Google Cloud Storage.
SETTING UP YOUR GOOGLE CLOUD PROJECT
Below are the steps you need to take on the Google Cloud Storage website to allow a Google Storage Platform project to interoperate with tools written for other cloud storage systems.
Select Project
- Go to the Google Cloud Storage website and sign in.
- Near the center of the page you’ll see a ‘Go to console’ button. Click on that.
- At the top of your screen, click the triangle to choose a project.
- This invokes a window for you to select a project. Here, if you wish, you may create a new project, too.
- For this example, ‘Accounting Documents’ is chosen by clicking on the link.
- You are immediately taken to your project dashboard.
Get the Request Endpoint URL
- Find the ‘Resources’ card/section. Click on ‘Storage.’ If you don’t see this, scroll down to the ‘Getting Started’ card and find and click on ‘Create a Cloud Storage bucket.’
- This takes you to the Storage page where your bucket(s) are listed.
- On the left sidebar, click on ‘Settings.’
- At the top, under the ‘Settings’ heading, click on ‘Interoperability.’ The settings within the Interoperability API allow Google Cloud Storage to interoperate with tools written for other cloud storage system.
- In the center of the page, you’ll see the ‘https://storage.googleapis.com‘ URL for the ‘Request endpoint.’ Make note of this (copy/paste somewhere) because you’ll need it when creating your connection profile in ChronoSync.
Set Up Access Keys
- Scroll to the bottom of this page and take a look at ‘User account HMAC’ (keyed-hash message authentication codes).
- Under ‘Default project for interoperable access,’ set the chosen project as the default.
- Under ‘Access keys for your user account,’ click on ‘Create A Key.’
- An Access key and Secret are created for you. Make of note of these (copy/paste somewhere) because you’ll need them both when creating your connection profile in ChronoSync.
HOW TO SET UP CHRONOSYNC TO CONNECT TO GOOGLE
- Open ChronoSync.
- Open Preferences.
- Go to ‘Connections.’
- Click on the ‘+’ to create a new connection profile.
- Name your profile and choose ‘Amazon S3’ as your connection type.
- For ‘Endpoint,’ use the ‘https://storage.googleapis.com’ URL.
- Copy in the Access Key and Secret that you just received while in the Google Storage settings.
- Now test the connection. Assuming everything went well, it should connect.
- Finish the connection profile setup. In the ‘Advanced’ panel, set ‘Multi-part upload block size’ to ‘150 MiB.’
- Click ‘Save.’
- Once saved, you can use this Amazon connection to sync or backup your Google Cloud buckets!
SET UP A SYNCHRONIZER TASK
- Create a new synchronizer task.
- Choose your targets, with one being your newly created Amazon connection.
- Click on ‘Choose…’ and select the bucket you wish to sync or backup to.
- Click on ‘Select.’
- You are now ready to sync to your Google Cloud bucket via an Amazon connection profile!
CAVEATS
Since Amazon S3 storage and Google Cloud storage use different techniques for keeping track of your data, there are some caveats when using Google Cloud via S3 emulation.
Metadata
Google Cloud’s S3 emulation does not support the creation of an object named “.” and thus cannot retain exotic metadata like extended attributes and resource forks. To get around this, you will need to turn OFF all the “Preserve…” file handling options (Comments, EAs, ACLs and Resource Forks) in the Options->Special File/Folder Handling section for it to even work.
When you make the above changes, you’ll get readiness warnings in the Setup panel saying both targets fully support preservation. This is not true! You must “ignore” those readiness warnings.
Because basic metadata is handled differently between S3 and Google Cloud, existing files will appear to have modification dates equal to the time they originally uploaded to Google Cloud. If you are doing a standard backup, this won’t be a problem. Those files will likely post-date the original files and won’t be replaced.
Because of the above problem, if you are doing a mirror, EVERY file will likely want to be replaced. On the plus side, all of the basic metadata will be updated. On the minus side, you’ll have to wait for everything to upload again. If this is undesirable, you can always perform a baseline scan and only new changes will be mirrored from here on.
COOL TIP: To bring basic metadata in sync, you can use the Validator. After disabling the “Preserve…” settings mentioned above, invoke the Validator (“Validate…” on the toolbar) and click “Recommended Settings,” if it appears. Then enable “Mark mismatched items for synchronization” and run the Validator. When the data contents of two files are the same, but modification dates differ, the Validator will only mark the item for an attribute sync. This will bring all the timestamps in sync with each other! NOTE: This could be a time-consuming process.
Encrypted Files
TECHNICAL SUPPORT
For most people, the above steps will work flawlessly for setting up an Amazon S3 connection to access your Google Cloud Storage. However, we live in an imperfect world and sometimes things just don’t work out as expected. If you encounter any problems, contact our technical support team and just ask! We don’t mind — we’re here to help!