- 2774 Views
- 3 replies
- 0 kudos
Avro file format generation
Hi All,We are using cluster with 9.1 run time version, I'm getting "incompatible schema exception" error while writing the data into avro file. Fields in Avro schema are more compared to dataframe output Fields. I tried the same in community edition ...
- 2774 Views
- 3 replies
- 0 kudos
- 2361 Views
- 1 replies
- 1 kudos
Auto loader
Suppose I have 1000's of historical .csv files stored from Jan, 2022 in a folder of my azure blob storage container. I want to use auto loader to read files beginning only on 1st, Oct, 2023 and ignoring all the files before this date to build a pipel...
- 2361 Views
- 1 replies
- 1 kudos
- 1 kudos
@BhaveshPatel Three things that you can do:- Move the files to the separate folder,- Use a filter on metadata fields to filter out the unnecessary files,- Use a pathGlobFilter to filter only on the files you need
- 1 kudos
- 3339 Views
- 3 replies
- 0 kudos
Python UDF fails with UNAVAILABLE: Channel shutdownNow invoked
I'm using a Python UDF to apply OCR to each row of a dataframe which contains the URL to a PDF document. This is how I define my UDF: def extract_text(url: str): ocr = MyOcr(url) extracted_text = ocr.get_text() return json.dumps(extracte...
- 3339 Views
- 3 replies
- 0 kudos
- 0 kudos
@Bharathi7 It's really hard to determine what's going on without knowing what acutally MyOcr function does.Maybe there's some kind of timeout on the service side? To many parallell connections?
- 0 kudos
- 3154 Views
- 1 replies
- 0 kudos
com.databricks.sql.transaction.tahoe.ColumnMappingException: Found duplicated column id `2` in colum
Hi,Currently, I am using the below-mentioned query to create a materialized view. It was working fine until yesterday in the DLT pipeline, but from today on, the below-provided code throws an error (com.databricks.sql.transaction.tahoe.ColumnMappingE...
- 3154 Views
- 1 replies
- 0 kudos
- 32572 Views
- 3 replies
- 2 kudos
Data type length enforcement
Hello. Is there a way to enforce the length of a column in SQL? For example that a column has to be exactly 18 characters? Thank you!
- 32572 Views
- 3 replies
- 2 kudos
- 2 kudos
we are facing similar issues while write into adls location delta format, after that we created on top delta location unity catalog tables. below format of data type length should be possible to change spark sql supported ?Azure SQL Spark ...
- 2 kudos
- 1411 Views
- 0 replies
- 0 kudos
Can i get 2 different type of Source file (csv and Json) in a single build
Need to understand this i got 2 sources which currently writes to the Single S3 bucket in csv and json format daily is it possible to ingest both the file types in a single build to ingest in Databricks, file size <300 mb ?or Alternately i should cre...
- 1411 Views
- 0 replies
- 0 kudos
- 3149 Views
- 2 replies
- 7 kudos
docs.databricks.com
Rename and drop columns with Delta Lake column mapping. Hi all,Now databricks started supporting column rename and drop.Column mapping requires the following Delta protocols:Reader version 2 or above.Writer version 5 or above.Blog URL##Available in D...
- 3149 Views
- 2 replies
- 7 kudos
- 7 kudos
Above mentioned feature is not working in the DLT pipeline. if the scrip has more than 4 columns
- 7 kudos
- 5700 Views
- 3 replies
- 0 kudos
Resolved! CLONE not supported on delta table with Liquid Clustering
Hello all,We are building a data warehouse on Unity Catalog and we use the SHALLOW CLONE command to allow folks to spin up their own dev environments by light copying the prod tables. We also started using Liquid Clustering on our feature tables, tho...
- 5700 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks Kaniz for your reply. I was able to get it make it work using runtime 14.0.Regards,
- 0 kudos
- 4385 Views
- 0 replies
- 0 kudos
Delta Lake Spark fails to write _delta_log via a Notebook without granting the Notebook data access
I have set up a Jupyter Notebook w/ PySpark connected to a Spark cluster, where the Spark instance is intended to perform writes to a Delta table.I'm observing that the Spark instance fails to complete the writes if the Jupyter Notebook doesn't have ...
- 4385 Views
- 0 replies
- 0 kudos
- 2276 Views
- 0 replies
- 0 kudos
Can I generate a uuid4 column when I do a COPY INTO command?
I have raw call log data and the logs don't have a unique id number so I generate a uuid4 number when i load them using spark. Now I want to save the records to a table, and run a COPY INTO command every day to ingest new records. I am only appendi...
- 2276 Views
- 0 replies
- 0 kudos
- 39707 Views
- 5 replies
- 1 kudos
How can I view an exported DBC notebook in my computer?
Is it possible to convert to or export as a .ipynb notebook?
- 39707 Views
- 5 replies
- 1 kudos
- 1 kudos
You can rename somefile.dbc to somefile.zip and open it with the Windows File Explorer, however the .python files cannot be opened in vscode or pycharm
- 1 kudos
- 2604 Views
- 2 replies
- 1 kudos
Not able to find the DBAcademyDLT policy to create the DLT Pipeline
Hi,Greetings of the day, I am preparing for the Databricks Engineering Associate Certification and following the Databricks Academy V3 course. As part of this course, trying to create the DLT pipelines but couldn't able to see the DBAcademyDLT policy...
- 2604 Views
- 2 replies
- 1 kudos
- 1 kudos
I had to go to Admin Settings > Advanced (Workspace settings) and enable "Cluster, Pool and Jobs Access Control". Afterwards I was able to create an instance pool.
- 1 kudos
- 2757 Views
- 1 replies
- 0 kudos
Hello unable to start using the notebook
I get this error when I run the Databricks training notebook%run ../Includes/Classroom-Setup-01 Resetting the learning environment: | dropping the catalog "***_53sh_da"...(0 seconds) Skipping install of existing datasets to "dbfs:/mnt/dbacademy-da...
- 2757 Views
- 1 replies
- 0 kudos
- 0 kudos
Could you fixed it? I have the same problem
- 0 kudos
- 1400 Views
- 0 replies
- 0 kudos
how to reduce file size in sedona o/p
I have shape files with polygon/geometry info. I am exporting the file after Sedona integration with Kepler.I o/p file is in .html. I want to reduce the file size.Pls suggest in case any option is available.
- 1400 Views
- 0 replies
- 0 kudos
- 1477 Views
- 0 replies
- 0 kudos
workspace deletion at Databricks recovery
Hi Team,I accidentally deleted our databricks workspace, which had all our artefacts and control plane, and was the primary resource for our team's working environment.Could anyone please help on priority, regarding the recovery/ restoration mechanis...
- 1477 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
