- 3418 Views
- 3 replies
- 1 kudos
Billing support contact
Hi,What is the best way to contact Databricks support? I see the differences between AWS billing and Databricks report and I'm looking for explanation of that. I've send 3 messages last week by this form https://www.databricks.com/company/contact but...
- 3418 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, I'm facing the same issue with signing in my workspace, and I have a billing error, databricks charged me for a free trial, and I have sent a lot of emails, posted a topic in the community, I contacted people in AWS and they said that it must be ...
- 1 kudos
- 17219 Views
- 10 replies
- 19 kudos
Resolved! Debugging!
Hi ML folks, We are using Databricks to train deep learning models. The code, however, has a complex structure of classes. This would work fine in a perfect bug-free world like Alice in Wonderland. Debugging in Databricks is awkward. We ended up do...
- 17219 Views
- 10 replies
- 19 kudos
- 19 kudos
Has this been solved yet; a mature way to debug code on databricks. I'm running in the same kind of issue.Variable explorer can be used and pdb, but not the same really..
- 19 kudos
- 7192 Views
- 2 replies
- 2 kudos
Resolved! How big should a delta table be to benefit from liquid clustering?
My questions is pretty straightforward - how big should a delta table be to benefit from liquid clustering? I know the answer will most likely depend on the details of how you are querying the data, but what is the recommendation?I know Databricks re...
- 7192 Views
- 2 replies
- 2 kudos
- 2 kudos
@DatBoi Once you watch this video you'll understand more about Liquid Clustering https://www.youtube.com/watch?v=5t6wX28JC_M&ab_channel=DeltaLakeLong story short:I know Databricks recommends not partitioning on tables less than 1 TB and aim for 1 GB ...
- 2 kudos
- 1113 Views
- 0 replies
- 0 kudos
Tracking DBMS CDC
We're using Databricks to incrementally extract data from SQL Server tables into S3. The data contains a timestamp column. We need a place to store the maximum retrieved timestamp per table so it can retrieved during the next run.Does Databricks cont...
- 1113 Views
- 0 replies
- 0 kudos
- 2529 Views
- 3 replies
- 0 kudos
Avro file format generation
Hi All,We are using cluster with 9.1 run time version, I'm getting "incompatible schema exception" error while writing the data into avro file. Fields in Avro schema are more compared to dataframe output Fields. I tried the same in community edition ...
- 2529 Views
- 3 replies
- 0 kudos
- 2229 Views
- 1 replies
- 1 kudos
Auto loader
Suppose I have 1000's of historical .csv files stored from Jan, 2022 in a folder of my azure blob storage container. I want to use auto loader to read files beginning only on 1st, Oct, 2023 and ignoring all the files before this date to build a pipel...
- 2229 Views
- 1 replies
- 1 kudos
- 1 kudos
@BhaveshPatel Three things that you can do:- Move the files to the separate folder,- Use a filter on metadata fields to filter out the unnecessary files,- Use a pathGlobFilter to filter only on the files you need
- 1 kudos
- 2823 Views
- 3 replies
- 0 kudos
Python UDF fails with UNAVAILABLE: Channel shutdownNow invoked
I'm using a Python UDF to apply OCR to each row of a dataframe which contains the URL to a PDF document. This is how I define my UDF: def extract_text(url: str): ocr = MyOcr(url) extracted_text = ocr.get_text() return json.dumps(extracte...
- 2823 Views
- 3 replies
- 0 kudos
- 0 kudos
@Bharathi7 It's really hard to determine what's going on without knowing what acutally MyOcr function does.Maybe there's some kind of timeout on the service side? To many parallell connections?
- 0 kudos
- 2996 Views
- 1 replies
- 0 kudos
com.databricks.sql.transaction.tahoe.ColumnMappingException: Found duplicated column id `2` in colum
Hi,Currently, I am using the below-mentioned query to create a materialized view. It was working fine until yesterday in the DLT pipeline, but from today on, the below-provided code throws an error (com.databricks.sql.transaction.tahoe.ColumnMappingE...
- 2996 Views
- 1 replies
- 0 kudos
- 29047 Views
- 3 replies
- 2 kudos
Data type length enforcement
Hello. Is there a way to enforce the length of a column in SQL? For example that a column has to be exactly 18 characters? Thank you!
- 29047 Views
- 3 replies
- 2 kudos
- 2 kudos
we are facing similar issues while write into adls location delta format, after that we created on top delta location unity catalog tables. below format of data type length should be possible to change spark sql supported ?Azure SQL Spark ...
- 2 kudos
- 1296 Views
- 0 replies
- 0 kudos
Can i get 2 different type of Source file (csv and Json) in a single build
Need to understand this i got 2 sources which currently writes to the Single S3 bucket in csv and json format daily is it possible to ingest both the file types in a single build to ingest in Databricks, file size <300 mb ?or Alternately i should cre...
- 1296 Views
- 0 replies
- 0 kudos
- 2850 Views
- 2 replies
- 7 kudos
docs.databricks.com
Rename and drop columns with Delta Lake column mapping. Hi all,Now databricks started supporting column rename and drop.Column mapping requires the following Delta protocols:Reader version 2 or above.Writer version 5 or above.Blog URL##Available in D...
- 2850 Views
- 2 replies
- 7 kudos
- 7 kudos
Above mentioned feature is not working in the DLT pipeline. if the scrip has more than 4 columns
- 7 kudos
- 5258 Views
- 3 replies
- 0 kudos
Resolved! CLONE not supported on delta table with Liquid Clustering
Hello all,We are building a data warehouse on Unity Catalog and we use the SHALLOW CLONE command to allow folks to spin up their own dev environments by light copying the prod tables. We also started using Liquid Clustering on our feature tables, tho...
- 5258 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks Kaniz for your reply. I was able to get it make it work using runtime 14.0.Regards,
- 0 kudos
- 4165 Views
- 0 replies
- 0 kudos
Delta Lake Spark fails to write _delta_log via a Notebook without granting the Notebook data access
I have set up a Jupyter Notebook w/ PySpark connected to a Spark cluster, where the Spark instance is intended to perform writes to a Delta table.I'm observing that the Spark instance fails to complete the writes if the Jupyter Notebook doesn't have ...
- 4165 Views
- 0 replies
- 0 kudos
- 2177 Views
- 0 replies
- 0 kudos
Can I generate a uuid4 column when I do a COPY INTO command?
I have raw call log data and the logs don't have a unique id number so I generate a uuid4 number when i load them using spark. Now I want to save the records to a table, and run a COPY INTO command every day to ingest new records. I am only appendi...
- 2177 Views
- 0 replies
- 0 kudos
- 37753 Views
- 5 replies
- 1 kudos
How can I view an exported DBC notebook in my computer?
Is it possible to convert to or export as a .ipynb notebook?
- 37753 Views
- 5 replies
- 1 kudos
- 1 kudos
You can rename somefile.dbc to somefile.zip and open it with the Windows File Explorer, however the .python files cannot be opened in vscode or pycharm
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
