- 4035 Views
- 3 replies
- 0 kudos
Writing to Snowflake from Databricks - sqlalchemy replacement?
I am trying to migrate some complex python load processes into databricks. Our load processes currently use pandas and we're hoping to refactor into Spark soon. For now, I need to figure out how to alter our functions that get sqlalchemy connection e...
- 4035 Views
- 3 replies
- 0 kudos
- 0 kudos
@brian999 - spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples. https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake Please let us know if this hel...
- 0 kudos
- 3401 Views
- 1 replies
- 0 kudos
TaskSensor - check if task is succeded
Hi,I would like to check if the task within job is succeded (even the job is marked as failed because on of the tasks).I need to create dependency for tasks within other jobs. The case is that I have one job for loading all tables for one country. Re...
- 3401 Views
- 1 replies
- 0 kudos
- 3384 Views
- 3 replies
- 0 kudos
How to cache on 500 billion rows
Hello!I'm using a server less SQL cluster on Data bricks and I have a dataset on Delta Table that has 500 billion rows. I'm trying to filter to have around 7 billion and the cache that dataset to use it on other queries and make it run faster.When I ...
- 3384 Views
- 3 replies
- 0 kudos
- 0 kudos
I missed the 'serverless sql' part. CACHE is for spark, I don´t think it works for serverless sql.Here is how caching works on DBSQL.
- 0 kudos
- 5119 Views
- 4 replies
- 0 kudos
COPY INTO skipRows FORMAT_OPTIONS does not work
Based on the COPY INTO documentation, it seems I can use `skipRows` to skip the first `n` rows. I am trying to load a CSV file where I need to skip a few first rows in the file. I have tried various combinations, e.g. setting header parameter on or ...
- 5119 Views
- 4 replies
- 0 kudos
- 0 kudos
@yubin-apollo: My bad - I had the skipRows in the COPY_OPTIONS and not in the FORMAT_OPTIONS. It works, please ignore my previous comment. Thanks
- 0 kudos
- 24094 Views
- 3 replies
- 4 kudos
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 188.0 failed 4
When I am trying to read the data from sql server through jdbc connect , I get the below error while merging the data into databricks table . Can you please help whats the issue related to? : org.apache.spark.SparkException: Job aborted due to stage...
- 24094 Views
- 3 replies
- 4 kudos
- 4 kudos
@rchauhan did you find a solution to the problem or know what settings caused the problem ?
- 4 kudos
- 3868 Views
- 4 replies
- 0 kudos
OPTIMIZE with liquid clustering makes filter slower than without OPTIMIZE
I created 15 Million records as a Delta Table and i'm running a simple filter query on that table based on one column value - which will return only one record. Because all the values on that column are unique.Delta Table is not partitioned.Before en...
- 3868 Views
- 4 replies
- 0 kudos
- 0 kudos
it seems that for this specific query Liquid Clustering has worse performance. It does not have better performance for all queries.The following are examples of scenarios that benefit from clustering:Tables often filtered by high cardinality columns...
- 0 kudos
- 6606 Views
- 2 replies
- 3 kudos
Module not found, despite it being installed on job cluster?
We observed the following error in a notebook which was running from a Databricks workflow: ModuleNotFoundError: No module named '<python package>'The error message speaks for itself - it obviously couldn't find the python package. What is peculiar ...
- 6606 Views
- 2 replies
- 3 kudos
- 3 kudos
Thanks, @Walter_C. Supposing that your second possible explanation, Cluster Initialization Timing, could be a factor, are there any best practices or recommendations for preventing this from being a recurring issue, down the road?
- 3 kudos
- 3014 Views
- 0 replies
- 0 kudos
CI/CD Databricks Asset Bundles - DLT pipelines - unity catalog and target schema
Is it possible for the CI/CD Databricks Asset Bundles YAML file to describe unity catalog and target schema as destination needed for the DLT pipeline? Or that's just not possible today.In case this functionality is not possible today, are there any ...
- 3014 Views
- 0 replies
- 0 kudos
- 4557 Views
- 2 replies
- 1 kudos
[FinOps] Tagging queries in databricks
Hello,I see that it is possible to tag catalogs/databases/tables. But I did not find a way to tag a query for our finop use case.In Azure you can check billings dependings on tags.A concrete example: In Azure Machine Learning, I have a schedule that ...
- 4557 Views
- 2 replies
- 1 kudos
- 4809 Views
- 2 replies
- 1 kudos
Delta Live Table - Cannot redefine dataset
Hi,I am new to Delta Live Table.I am trying to create a delta live table from the databricks tutorial.I have created a notebook and attached an interactive cluster -DBR 14.3-LTS.I am running the below code.When I ran it for the 1st time it ran succes...
- 4809 Views
- 2 replies
- 1 kudos
- 1 kudos
The error message "AnalysisException: Cannot redefine dataset 'sales_orders_raw'" is indicating that you're trying to create a table that already exists. In Databricks, once a Delta Live Table (DLT) is defined, it cannot be redefined or overwritten. ...
- 1 kudos
- 3230 Views
- 0 replies
- 0 kudos
Driver is up but is not responsive, likely due to GC.
I am using Interactive cluster to run frequent (every 15min) batch job.After certain time (example: 6hours), the cluster continuously starts showing Driver is up but is not responsive, likely due to GC. in event log and all jobs starts failing.If the...
- 3230 Views
- 0 replies
- 0 kudos
- 4006 Views
- 1 replies
- 0 kudos
Consume updated data from the Materialized view and send it as append to a streaming table
Hello everyone! I'm using DLT and I'm pretty new to them. I'm trying to take the updates from a materialized view and send them to a streaming table as an append.For example, if I have a MV of 400 records, I want an append to be made to the streaming...
- 4006 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @WearBeard By default, streaming tables require append-only sources. The encountered error is due to an update or delete operation on the 'streaming_table_test'. To fix this issue, perform a Full Refresh on the 'streaming_table_test' table. You ca...
- 0 kudos
- 2134 Views
- 0 replies
- 0 kudos
Not able to create mount point in Databricks
HiI am trying to create mount point in Azure Databricks, but mount point creation is failing with below error messageDBUtils.FS Handler.mount() got an unexpected keyword argument 'extra_config'I am using following codedef setup_mount(storage_account_...
- 2134 Views
- 0 replies
- 0 kudos
- 1876 Views
- 1 replies
- 0 kudos
Exam got suspended without any reason
Hi Team,My Databricks Certified Associate Developer for Apache Spark 3.0 - Python exam got suspended on 7th March 2024I was there continuously in front of the camera and suddenly the alert appeared, and support person asked me to show the full table ...
- 1876 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @badari_narayan did you exam got rescheduled..i am also facing same issue my exam got suspemded
- 0 kudos
- 3438 Views
- 2 replies
- 0 kudos
Fail to install package dependency located on private pypi server during .whl installation
Hello,I recently switched from DBR 12.2 LTS to DBR 13.3 LTS and observed the following behavior:My goal is to install a python library from a .whl file. I am using the UI for this task (Cluster settings -> Libraries -> Install new -> 'Python Whl' as ...
- 3438 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Felix, I have run into a similar issue recently (my wheel needs a Git HTTPS redirect that's specified in the init script - but I can install it fine from inside a notebook).I wonder whether you found a solution (perhaps moving a more recent DBR v...
- 0 kudos
| User | Count |
|---|---|
| 1638 | |
| 791 | |
| 536 | |
| 349 | |
| 287 |
