- 3334 Views
- 4 replies
- 0 kudos
Based on the COPY INTO documentation, it seems I can use `skipRows` to skip the first `n` rows. I am trying to load a CSV file where I need to skip a few first rows in the file. I have tried various combinations, e.g. setting header parameter on or ...
- 3334 Views
- 4 replies
- 0 kudos
Latest Reply
@yubin-apollo: My bad - I had the skipRows in the COPY_OPTIONS and not in the FORMAT_OPTIONS. It works, please ignore my previous comment. Thanks
3 More Replies
by
Teja07
• New Contributor II
- 7260 Views
- 2 replies
- 0 kudos
How to copy a file from local disk to databricks dbfs path. I tried as below but it is throwing me error:code i tried: dbutils.fs.cp("file://c:/user/file.txt",dbfs/data/) and dbutils.fs.cp("file:///c:/user/file.txt",dbfs/data/) error: File not found ...
- 7260 Views
- 2 replies
- 0 kudos
Latest Reply
I assume you cannot copy files from Local machine to dbfs using dbutils. you can upload files to dbfs using below gui option . Data --> Browse DFS --> Upload
1 More Replies
- 4199 Views
- 1 replies
- 1 kudos
Arrays of complex types seemingly always evaluate to ARRAY<STRING>. Therefore, casting or attempting to load JSON data with empty array values fails. For example, attempting to cast a JSON value of {"likes": []...} on load to the following table sche...
- 4199 Views
- 1 replies
- 1 kudos
Latest Reply
Hi @Jake Neyer Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1558 Views
- 1 replies
- 2 kudos
Hi, I am new to Databricks. I need to setup a non-prod environment for which I need data of prod to be cloned in non-prod. Explored some and got to know about shallow copy. Is it possible to do shallow copy across environments? or Is it possible to d...
- 1558 Views
- 1 replies
- 2 kudos
Latest Reply
@deepak prasad I'm not sure it's possible to do that. Even with Unity Catalog enabled, you cannot use shallow clone.You can do two things here:Without UC - just simply recreate an empty table in your non-prod environment and do SELECT * from prod st...
- 6579 Views
- 9 replies
- 9 kudos
I am trying to copy files from azure to s3. I've created a solution by comparing file lists and copy manually to a temp file and upload. However, I just found AutoLoader and I would like to use that https://docs.databricks.com/ingestion/auto-loader/i...
- 6579 Views
- 9 replies
- 9 kudos
Latest Reply
Just use tools like Goodsync and Gs Richcopy 360 to copy directly from blob to S3, I think you will never face problems like that
8 More Replies
- 8212 Views
- 6 replies
- 5 kudos
Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...
- 8212 Views
- 6 replies
- 5 kudos
- 5299 Views
- 6 replies
- 4 kudos
I would like to check if there is a process to copy a script/code or migrate the script from the current subscription of the Azure Databricks - Notebooks to new subscription of Databricks (new notebook).
- 5299 Views
- 6 replies
- 4 kudos
Latest Reply
how quickly move the Databricks notebooks from one account to another?
5 More Replies
by
hoopla
• New Contributor II
- 7063 Views
- 2 replies
- 1 kudos
I am downloading multiple files by web scraping and by default they are stored in /tmp
I can copy a single file by providing the filename and path
%fs cp file:/tmp/2020-12-14_listings.csv.gz dbfs:/tmp
but when I try to copy multiple files I get an ...
- 7063 Views
- 2 replies
- 1 kudos
Latest Reply
Thanks DeepakThis is what I have suspected.Hopefully the wild card feature might be available in futureThanks
1 More Replies
- 1461 Views
- 1 replies
- 0 kudos
It seems like with both techniques I would end up with a copy of my table. Trying to understand when I should be using a deep clone.
- 1461 Views
- 1 replies
- 0 kudos
Latest Reply
A deep clone is recommended way as it holds the history of the table. Also, the DEEP clone is faster than the read-write approach.