- 2339 Views
- 3 replies
- 0 kudos
how to create catalog
Hi ,I am trying to create catalog and database its not allowing databricks , please suggest .Here my code .base_dir = "/mnt/files"spark.sql(f"CREATE CATALOG IF NOT EXISTS dev")spark.sql(f"CREATE DATABASE IF NOT EXISTS dev.demo_db") first i ne...
- 2339 Views
- 3 replies
- 0 kudos
- 0 kudos
I got a similar error trying create a catalog with "databricks.sdk" library I resolved it add the parameter "storage_root": w.catalogs.create(name=c.name, storage_root='s3://databricks-workspace-bucket/unity-catalog/426335709') In my case all catalog...
- 0 kudos
- 1996 Views
- 1 replies
- 0 kudos
Resolved! Installing Maven (3rd party) libraries on Job Cluster
I'm trying to install Maven Libraries on the job cluster (non interactive cluster) as part of databricks workflow. I've added the context in the cluster configuration as part of deployment which I cant find the same in the post deployment configurati...
- 1996 Views
- 1 replies
- 0 kudos
- 0 kudos
I found the workaround. Below are the steps:1. Add the required library to the Allowed list at the workspace level (require workspace/metastore admin access); you might need coordinates groupdd:artifactId:version2. At the task level, include under De...
- 0 kudos
- 1013 Views
- 1 replies
- 0 kudos
Cluster configuration
Please help me configure/choose the cluster configuration. I need to process and merge 6 million records into Azure SQL DB. At the end of the week, 9 billion records need to be processed and merged into Azure SQL DB, and a few transformations need to...
- 1013 Views
- 1 replies
- 0 kudos
- 0 kudos
It will depend on the transformations and how you're loading them. Assuming it's mostly in spark, I recommend starting small using a job compute cluster with autoscaling enabled for cost efficiency. For daily loads (6 million records), a driver and 2...
- 0 kudos
- 1236 Views
- 1 replies
- 1 kudos
Resolved! Permission Issue in Delta Lake Course
Hi everyone,I'm new to Databricks and working on the "Data Ingestion with Delta Lake" course. I encountered a permission error with the following query:Can anyone help with this?Thanks!
- 1236 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @walgt! Apologies for the inconvenience. This was a known issue, but it has now been fixed! You should now be able to run your query without any problems. Thanks for your patience!
- 1 kudos
- 8405 Views
- 1 replies
- 0 kudos
Connecting to Azure PostgreSQL from Azure Databricks
Hello,In Databricks there are 2 ways to connect to PostgreSQL, i.e., using JDBC driver or the named connector as mentioned in the document - https://learn.microsoft.com/en-us/azure/databricks/external-data/postgresqlFor JDBC, the driver needs to be ...
- 8405 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kp12,I just wanted to check whether you found the answer or not.I also want to know the difference because the named connector "PostgreSQL" is overwhelming the CPU of PostgreSQL while inserting 41M rows.
- 0 kudos
- 1295 Views
- 1 replies
- 0 kudos
Choice of SQL Warehouse
Hi community,I am studying the documentation about different kind of SQL warehouse (https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-types#:~:text=A%20classic%20SQL%20warehouse%20supports,than%20in%20your%20Databricks%20account.)I s...
- 1295 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Brianben! Classic SQL warehouses are better for cost-sensitive 24/7 workloads, stable query patterns, and older workflows that depend on traditional data warehouse setups or external Hive metastores. They also allow some manual configuration, ...
- 0 kudos
- 3757 Views
- 9 replies
- 4 kudos
- 3757 Views
- 9 replies
- 4 kudos
- 4 kudos
@Shua42 I've realized that I didn’t mention the subquery issue in my first post, so I guess this can be handled as a separate ticket.
- 4 kudos
- 3065 Views
- 5 replies
- 0 kudos
Connect Databricks to Airflow
Hi,I have Databricks on top of aws. I have a Databricks connection on Airflow (mwaa). I am able to conect and execute a Datbricks job via Airflow using a personal access token. I believe the best practice is to conect using a service principal. I und...
- 3065 Views
- 5 replies
- 0 kudos
- 0 kudos
https://airflow.apache.org/docs/apache-airflow-providers-databricks/6.9.0/connections/databricks.html
- 0 kudos
- 6538 Views
- 3 replies
- 0 kudos
Unity Catalog - Storage Account Data Access
I was exploring on unity catalog option on Databricks premium workspace.I understood that i need to create storage account credentials and external connection in workspace.Later, i can access the cloud data using 'abfss://storage_account_details' .I ...
- 6538 Views
- 3 replies
- 0 kudos
- 0 kudos
Databricks strategic direction is to deprecate mount points in favor of Unity Catalog Volumes.Setup an STORAGE CREDENTIAL and EXTERNAL LOCATION to access and define how to get to your cloud storage account. To access data on the account, define a Tab...
- 0 kudos
- 5435 Views
- 4 replies
- 2 kudos
PyTest working in Repos but not in Databricks Asset Bundles
Hello,I am trying to run PyTest from a notebook or python file that exists due to being deployed by a Databricks Asset Bundle (DAB).I have a repository that contains a number of files with the end goal of trying to run PyTest in a directory to valida...
- 5435 Views
- 4 replies
- 2 kudos
- 2 kudos
@ChrisLawford You can run pytest through job databricks bundle run -t dev pytest_job I was able to work around in this way.resource/pytest.job.ymlresources: jobs: pytest_job: name: pytest_job tasks: - task_key: pytest_task ...
- 2 kudos
- 3923 Views
- 2 replies
- 0 kudos
How to migrate DBFS from one tenant to another tenant
I am working on Databricks workspace migration, where I need to copy the Databricks workspace including DBFS from source to target (both source and target are in different subscription/account). Can someone suggest what could be approach to migrate D...
- 3923 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jose_gonzalez How can we migrate the data from Managed storage account from azure data bricks to new tenant?
- 0 kudos
- 1129 Views
- 1 replies
- 1 kudos
Resolved! Change cluster owner API call failed
I am trying to change cluster's owner using API call. but get following error. I am positive that host, cluster_id and owner_username are all correct. The error message says No API found. Is this related with the compute I am using? or something else...
- 1129 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi hdu,How are you doing today?, As per my understanding, It sounds like you’re really close! That “No API found” error usually means either the wrong API endpoint is being used, or the cluster type doesn’t support ownership changes—for example, shar...
- 1 kudos
- 1990 Views
- 4 replies
- 3 kudos
What's the recommended way of creating tables in Databricks with unity catalog (External/Managed)
I have databricks with unity catalog enables and created an external ADLS location. when I create the catalog/schema it uses the external location. when I try to create the table it uses the extrenal location but they are managed tables. What's the r...
- 1990 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi Shivap,How are you doing today?, As per my understanding, in Unity Catalog, if you want to create an external table, you just need to make sure the external location is registered and approved first. Even though you're specifying a path with LOCAT...
- 3 kudos
- 3053 Views
- 8 replies
- 2 kudos
Resolved! When will DB release runtime with Scala 2.13
When will DB release runtime with Scala 2.13? Thx
- 3053 Views
- 8 replies
- 2 kudos
- 2 kudos
Spark 4 is coming and it's only going to support Scala 2.13, so a Databricks Runtime with Spark 3.5.x and Scala 2.13 should be released soonish.
- 2 kudos
- 3513 Views
- 1 replies
- 1 kudos
Job Retry Wait Policy and Cluster Shutdown
Hi all,I have a Databricks Workflow job in which the final task makes an external API call. Sometimes this API will be overloaded and the call will fail. In the spirit of automation, I want this task to retry the call an hour later if it fails in the...
- 3513 Views
- 1 replies
- 1 kudos
- 1 kudos
What about this? Any reply? Any alternative? I'm facing the same issue.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
