- 1177 Views
- 1 replies
- 1 kudos
Sending my weather data to a clients cloud storage
Hi, One of our clients is asking to switch from our API feed to have weather data delivered automatically to their Cloud Storage. What steps do I need to take from my end? Do I need to join Databricks to do so? Thanks. Tom
- 1177 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @TomHauf, while it may not be essential in your case, you should at least consider using Databricks to facilitate loading data into your customers cloud storage. Databricks gives you a few options to make sharing with third parties simple as per ...
- 1 kudos
- 5455 Views
- 2 replies
- 0 kudos
Can job 'run_as' be assigned to users/principals who actually run it?
Can job 'run_as' be assigned to users/principals who actually run it? instead of always a fixed creator/user/pricipal?When a job is run, I would like to see in the job setting "run_as" the name of the actual user/principal who runs it.Currently, "run...
- 5455 Views
- 2 replies
- 0 kudos
- 0 kudos
Yeah, the functionality is odd. You can't change the Run As user to anyone but yourself. But you can run it using the Run As setting previously used. This sort of makes sense if the workflow is created to be run as a service principal with specific p...
- 0 kudos
- 3093 Views
- 1 replies
- 0 kudos
CLOUD_PROVIDER_RESOURCE_STOCKOUT (Azure)
Hey All,Anyone ran into this 'out of stock' error on certain types of clusters? We've spent months building on Standard_D8ads_v5 (delta cache) and this morning a see of red because there are none available. I can't even spin up a small interactive...
- 3093 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi JJ, The CLOUD_PROVIDER_RESOURCE_STOCKOUT error code indicates that the cloud provider is out of physical capacity underneath virtual machines. The failure was caused by the cloud provider and I would recommend you reaching out to the respective c...
- 0 kudos
- 5087 Views
- 3 replies
- 0 kudos
Warehouse Name in System Tables
Hello.I am creating a table to monitor the usage of All-purpose Compute and SQL Warehouses. From the tables in 'system' catalog, I can get cluster_name and cluster_id. However only warehouse_id is available and not warehouse name. Is there a way to g...
- 5087 Views
- 3 replies
- 0 kudos
- 0 kudos
I have enabled the compute system schema, but i dont see this tabke. What am i missing?
- 0 kudos
- 2607 Views
- 4 replies
- 0 kudos
DAB - All projects files deployed
I have an issue with DAB where all the project files, starting from root ., get deployed to the /files folder in the bundle. I would prefer being able to deploy certain util notebooks, but not all the files of the project. I'm able to not deploy any ...
- 2607 Views
- 4 replies
- 0 kudos
- 0 kudos
@cmathieu , It will support deployment of whole directory and not others as well.
- 0 kudos
- 1860 Views
- 2 replies
- 0 kudos
Resolved! Error while reading file from Cloud Storage
The code we are executing: df = spark.read.format("parquet").load("/mnt/g/drb/HN/") df.write.mode('overwrite').saveAsTable("bronze.HN")the error it throws:org.apache.spark.SparkException: Job aborted due to stage failure: Task 44 in stage 642.0 faile...
- 1860 Views
- 2 replies
- 0 kudos
- 1375 Views
- 1 replies
- 0 kudos
Job , Task, Stage Creation
I am running below code -df = spark.read.json('xyz.json')df.countI want to understand the actual working of the spark. How many jobs & stages will be created. I want to understand the detailed & easier concept of how it works?
- 1375 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Rajt1! When you execute df = spark.read.json('xyz.json’), Spark does not read the file immediately. Data is only read when an action like count() is triggered. Job: df.count() triggers one job because it's an action.Stage: Reading JSON and cou...
- 0 kudos
- 2684 Views
- 3 replies
- 0 kudos
Error "insert concurrent to Delta Lake" when 2 streaming merge data to same table at the same time
Hello everyone ,We currently have 2 streaming (Bronze job) created on 2 tasks in the same job, running the same compute job and both merge data into the same table (Silver table). If I create it like above, sometimes I get an error related to "insert...
- 2684 Views
- 3 replies
- 0 kudos
- 4149 Views
- 2 replies
- 0 kudos
Resolved! Connecting to SQL on Databricks Using SQLAlchemy or pyodbc
On Databricks, when I try to connect to SQL using SQLAlchemy or pyodbc to run delete queries on a specific table, I get this error: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not ...
- 4149 Views
- 2 replies
- 0 kudos
- 0 kudos
I was able to solve the problem! the problem was because the driver was missing and so pyodbc or sqlAlchemy can't find it. So I used the native Java API and it is working.This is the example code:jdbcUsername = "username"jdbcPassword = "password"driv...
- 0 kudos
- 1262 Views
- 1 replies
- 1 kudos
Resolved! Unable to delete a table
When I try to delete a table, I'm getting this error:[ErrorClass=INVALID_STATE] TABLE catalog.schema.table_name cannot be deleted because it is being shared via Delta Sharing.I have checked on the internet about it, but could not find any info about ...
- 1262 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @IGRACH ,You are facing this issue bcz I guess the table you want to delete is being shared by delta sharing. You can go to the shared object by following this dochttps://docs.databricks.com/aws/en/delta-sharing/create-share#update-sharesandThen, ...
- 1 kudos
- 3086 Views
- 3 replies
- 1 kudos
External tables in DLT pipelines
Hello community,I have implemented a DLT pipeline.In the "Destination" setting of the pipeline I have specified a unity catalog with target schema of type external referring to an S3 destination.My DLT pipeline works well. Yet, I noticed that all str...
- 3086 Views
- 3 replies
- 1 kudos
- 1 kudos
This won't work.best approach is create dlt sink to write to delta external table. This pipeline should only be 1 step. Read table and append flow using data sink. It works fine.
- 1 kudos
- 1560 Views
- 1 replies
- 0 kudos
Resolved! databricks bundle Deploy: exit code 0 even if an error occurs
We have a CI/CD pipeline where we run:databricks bundle deploy [...]The code works fine, however, if we missconfigure it, we see in the output an error message such asDeploying resources... Updating deployment state... Warning: Detected unresolved va...
- 1560 Views
- 1 replies
- 0 kudos
- 0 kudos
you can close it: it was an ci/cd issue
- 0 kudos
- 7138 Views
- 6 replies
- 1 kudos
Custom docker image fails to initalize
I'm trying to use a custom docker image for my job. This is my docker file:FROM databricksruntime/standard:12.2-LTS COPY . . RUN /databricks/python3/bin/pip install -U pip RUN /databricks/python3/bin/pip install -r requirements.txt USER rootMy job ...
- 7138 Views
- 6 replies
- 1 kudos
- 1179 Views
- 1 replies
- 0 kudos
Having an issue assigning databricks_current_metastore with terraform provider
I am trying to assign my databricks_current_metastore on terraform and I get the following error back as an output Error: cannot read current metastore: cannot get client current metastore: invalid Databricks Workspace configurationwith data.databric...
- 1179 Views
- 1 replies
- 0 kudos
- 0 kudos
@badari_narayan Based on above terraform code, you are trying to use the databricks.accounts provider to read the current workspace metastore, which is incorrect — the databricks_current_metastore data source is a workspace-level resource, and must b...
- 0 kudos
- 16628 Views
- 5 replies
- 4 kudos
Azure Databricks: Error Creating Cluster
We have provisioned a new workspace in Azure using our own VNet. Upon creating the first cluster, I encounter this error:Control Plane Request Failure: Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instan...
- 16628 Views
- 5 replies
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
