- 1443 Views
- 1 replies
- 0 kudos
Unable to create Cluster via REST API
I have tried all 3 of the example payloads listed here to create a Compute Cluster: https://docs.databricks.com/api/azure/workspace/clusters/createAll return the same error: Invalid JSON given in the body of the request - expected a mapOther Compute ...
- 1443 Views
- 1 replies
- 0 kudos
- 0 kudos
@jeffn - I was able to successfully create a cluster using the same Azure documentation, both via the REST API and the Databricks SDK. Hopefully it helps.from databricks.sdk import WorkspaceClient from databricks.sdk.service.compute import CreateClu...
- 0 kudos
- 1383 Views
- 1 replies
- 0 kudos
Resolved! Regexp_replaces pattern only once
select regexp_replace('asdfhsdf&&1&&asdfasdf&&2&&asdf','&&[0-100]&&','') output here it replaces first pattern but not the 2nd patterno/poutputasdfhsdfasdfasdf&&2&&asdf even if we use position its not working something like both wont work select ...
- 1383 Views
- 1 replies
- 0 kudos
- 0 kudos
changing to [0-9] works this can be closed.
- 0 kudos
- 14043 Views
- 3 replies
- 2 kudos
Resolved! CREATE TEMP TABLE
The Databricks assistant tells me (sometimes) that `CREATE TEMP TABLE` is a valid SQL operation. And other sources (e.g., https://www.freecodecamp.org/news/sql-temp-table-how-to-create-a-temporary-sql-table/) say the same.But in actual practice, thi...
- 14043 Views
- 3 replies
- 2 kudos
- 2 kudos
In addition to accepting KaranamS's answer, I will note a longer and useful discussion, with caveats, at https://community.databricks.com/t5/data-engineering/how-do-temp-views-actually-work/m-p/20137/highlight/true#M13584.
- 2 kudos
- 3442 Views
- 5 replies
- 0 kudos
Remote SQL Server Instance Connection using JDBC
Hi All,I am connecting Remote SQL Server Instance using JDBC Driver, I have enabled TCP/IP and Setup the Firewall Rule. When i am querying the instance i am getting this error:(com.microsoft.sqlserver.jdbc.SQLServerException) The TCP/IP connection to...
- 3442 Views
- 5 replies
- 0 kudos
- 0 kudos
If you want to access a local SQL server, you'll need a Private Link to access the server. If it's on your own local machine, that's likely not possible. Creating a VPN to your machine is a unique problem, and you would be better off using a VM or a ...
- 0 kudos
- 8441 Views
- 3 replies
- 7 kudos
what formatter is used to format SQL cell in databricks
Databricks launched formatter Black which formats python cells, I can also see SQL cell getting formatted, but not sure which formatter is being used for SQL cell formatting. No clarity given on docs.
- 8441 Views
- 3 replies
- 7 kudos
- 7 kudos
The formatter is mentioned here: Develop code in Databricks notebooks - Azure Databricks | Microsoft LearnIt is this npm package: @gethue/sql-formatter - npm
- 7 kudos
- 6315 Views
- 8 replies
- 3 kudos
Resolved! How to copy notebooks from local to the tarrget folder via asset bundles
Hi all,I am able to deploy Databricks assets to the target workspace. Jobs and workflows can also be created successfully.But I have aspecial requirement, that I copy the note books to the target folder on databricks workspace.Example:on Local I have...
- 6315 Views
- 8 replies
- 3 kudos
- 3 kudos
What are the permissions to this databricks directory? Can someone delete this directory or any file? On Shared workspace everyone can delete bundle files or bundle directory, even if in databricks.yml I provided permissions only to admins ('CAN MANA...
- 3 kudos
- 1249 Views
- 1 replies
- 1 kudos
Sending my weather data to a clients cloud storage
Hi, One of our clients is asking to switch from our API feed to have weather data delivered automatically to their Cloud Storage. What steps do I need to take from my end? Do I need to join Databricks to do so? Thanks. Tom
- 1249 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @TomHauf, while it may not be essential in your case, you should at least consider using Databricks to facilitate loading data into your customers cloud storage. Databricks gives you a few options to make sharing with third parties simple as per ...
- 1 kudos
- 5664 Views
- 2 replies
- 0 kudos
Can job 'run_as' be assigned to users/principals who actually run it?
Can job 'run_as' be assigned to users/principals who actually run it? instead of always a fixed creator/user/pricipal?When a job is run, I would like to see in the job setting "run_as" the name of the actual user/principal who runs it.Currently, "run...
- 5664 Views
- 2 replies
- 0 kudos
- 0 kudos
Yeah, the functionality is odd. You can't change the Run As user to anyone but yourself. But you can run it using the Run As setting previously used. This sort of makes sense if the workflow is created to be run as a service principal with specific p...
- 0 kudos
- 3337 Views
- 1 replies
- 0 kudos
CLOUD_PROVIDER_RESOURCE_STOCKOUT (Azure)
Hey All,Anyone ran into this 'out of stock' error on certain types of clusters? We've spent months building on Standard_D8ads_v5 (delta cache) and this morning a see of red because there are none available. I can't even spin up a small interactive...
- 3337 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi JJ, The CLOUD_PROVIDER_RESOURCE_STOCKOUT error code indicates that the cloud provider is out of physical capacity underneath virtual machines. The failure was caused by the cloud provider and I would recommend you reaching out to the respective c...
- 0 kudos
- 5220 Views
- 3 replies
- 0 kudos
Warehouse Name in System Tables
Hello.I am creating a table to monitor the usage of All-purpose Compute and SQL Warehouses. From the tables in 'system' catalog, I can get cluster_name and cluster_id. However only warehouse_id is available and not warehouse name. Is there a way to g...
- 5220 Views
- 3 replies
- 0 kudos
- 0 kudos
I have enabled the compute system schema, but i dont see this tabke. What am i missing?
- 0 kudos
- 2840 Views
- 4 replies
- 0 kudos
DAB - All projects files deployed
I have an issue with DAB where all the project files, starting from root ., get deployed to the /files folder in the bundle. I would prefer being able to deploy certain util notebooks, but not all the files of the project. I'm able to not deploy any ...
- 2840 Views
- 4 replies
- 0 kudos
- 0 kudos
@cmathieu , It will support deployment of whole directory and not others as well.
- 0 kudos
- 2041 Views
- 2 replies
- 0 kudos
Resolved! Error while reading file from Cloud Storage
The code we are executing: df = spark.read.format("parquet").load("/mnt/g/drb/HN/") df.write.mode('overwrite').saveAsTable("bronze.HN")the error it throws:org.apache.spark.SparkException: Job aborted due to stage failure: Task 44 in stage 642.0 faile...
- 2041 Views
- 2 replies
- 0 kudos
- 1495 Views
- 1 replies
- 0 kudos
Job , Task, Stage Creation
I am running below code -df = spark.read.json('xyz.json')df.countI want to understand the actual working of the spark. How many jobs & stages will be created. I want to understand the detailed & easier concept of how it works?
- 1495 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Rajt1! When you execute df = spark.read.json('xyz.json’), Spark does not read the file immediately. Data is only read when an action like count() is triggered. Job: df.count() triggers one job because it's an action.Stage: Reading JSON and cou...
- 0 kudos
- 2808 Views
- 3 replies
- 0 kudos
Error "insert concurrent to Delta Lake" when 2 streaming merge data to same table at the same time
Hello everyone ,We currently have 2 streaming (Bronze job) created on 2 tasks in the same job, running the same compute job and both merge data into the same table (Silver table). If I create it like above, sometimes I get an error related to "insert...
- 2808 Views
- 3 replies
- 0 kudos
- 4299 Views
- 2 replies
- 0 kudos
Resolved! Connecting to SQL on Databricks Using SQLAlchemy or pyodbc
On Databricks, when I try to connect to SQL using SQLAlchemy or pyodbc to run delete queries on a specific table, I get this error: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not ...
- 4299 Views
- 2 replies
- 0 kudos
- 0 kudos
I was able to solve the problem! the problem was because the driver was missing and so pyodbc or sqlAlchemy can't find it. So I used the native Java API and it is working.This is the example code:jdbcUsername = "username"jdbcPassword = "password"driv...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
