- 27368 Views
- 5 replies
- 4 kudos
How can I connect to an Azure SQL db from a Databricks notebook?
I know how to do it with spark, and read/write tables (like https://docs.microsoft.com/en-gb/azure/databricks/data/data-sources/sql-databases#python-example )But this time, I need to only update a field of a specific row in a table. I do not think I ...
- 27368 Views
- 5 replies
- 4 kudos
- 4 kudos
Look at your compute configuration. Looks like this works perfectly on Dedicated (formerly: single user) or No isolation clusters, but not on Standard (formerly: Shared) ones.Maybe you need a disposable one-time job cluster with these settings.
- 4 kudos
- 2649 Views
- 1 replies
- 0 kudos
Query limiting to only 10000 rows
Hi I am query my Azure Databricks table using VB script/Simba Spark ODBC driver.While querying into the DB(lets say 'Select * from table_1') it is not returning any data. However while querying a limit (lets say 'Select TOP 10000 ID from table_1'), i...
- 2649 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Y2KEngineer This issue is likely related to row fetching limits or buffer sizes in the driver or system settings. You can try adjusting a couple of things in your ODBC connection string:Set RowsFetchedPerBlock=50000 to make sure it fetches all ro...
- 0 kudos
- 1961 Views
- 2 replies
- 0 kudos
Pipelines are expected to have at least one table Error While running DLT pipeline
Error :Pipelines are expected to have at least one table defined butno tables were found in your pipeline I wrote simple code as phase 1 debug%sql CREATE OR REFRESH STREAMING TABLE test_table AS SELECT "hello" as greeting; Can u plz help what's wrong...
- 1961 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @vignesh22 - Adding to what @Takuya-Omi san has mentioned - the instantiation of streaming table in your definition is incorrect. You're trying to create a stream table using a batch source which will result in the DLT Analysis Exception as descr...
- 0 kudos
- 1280 Views
- 1 replies
- 2 kudos
Community Edition Data recovery
I was using Databricks Community Edition for some practice work, and I had created a few notebooks as part of my learning. However, when I recently tried to log in, I received a message saying, "We were not able to find a Community Edition." Now, non...
- 1280 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi saikrishna1020,How are you doing today? I totally understand how upsetting it can be to lose work you’ve put effort into. With Databricks Community Edition, unfortunately, inactivity for an extended period (usually 14–30 days) can cause the worksp...
- 2 kudos
- 1206 Views
- 1 replies
- 0 kudos
INVALID_PARAMETER_VALUE.LOCATION_OVERLAP when trying to copy from s3 location
Hi,Currently we are getting an issue when we try to copy a file from s3 location using dbutils.fs.cp, please see example below:source = s3://test-bucket/external/zones/{database_name}/{table_name}/test.csvdestination = s3://test-bucket/external/desti...
- 1206 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi dplatform_user,How are you doing today?, As per my understanding, this error is actually a common one when working with external storage paths that overlap with Unity Catalog-managed locations. The error message is basically saying that your sourc...
- 0 kudos
- 21177 Views
- 4 replies
- 0 kudos
Resolved! How to enforce delta table column to have unique values?
Hi,I have defined a delta table with a primary key:%sql CREATE TABLE IF NOT EXISTS test_table_pk ( table_name STRING NOT NULL, label STRING NOT NULL, table_location STRING NOT NULL, CONSTRAINT test_table_pk_col PRIMARY KEY(table_name) ...
- 21177 Views
- 4 replies
- 0 kudos
- 0 kudos
CREATE TABLE table_name (id_col1 BIGINT GENERATED ALWAYS AS IDENTITY,id_col2 BIGINT GENERATED ALWAYS AS IDENTITY (START WITH -1 INCREMENT BY 1),id_col3 BIGINT GENERATED BY DEFAULT AS IDENTITY,id_col4 BIGINT GENERATED BY DEFAULT AS IDENTITY (START WIT...
- 0 kudos
- 4735 Views
- 3 replies
- 6 kudos
Resolved! SERVERLESS SQL WAREHOUSE
Hello All,I have two questions regarding the serverless SQL warehouse which are following:1.) If I create a small Serverless SQL Warehouse in Databricks that shows 12 DBUs/hour, will I be charged 12 DBUs even if I don’t run any queries in that hour? ...
- 4735 Views
- 3 replies
- 6 kudos
- 6 kudos
Shua42 hits the nail on the head. If I can be so bold as to summarize: You are only charged when the Warehouse is running regardless of how much or how little you use it. We do have an auto stop feature you can configure. Essentially, you set a time...
- 6 kudos
- 16016 Views
- 3 replies
- 1 kudos
%pip install requirements.txt - path not found
Hi everyone,I was just testing things out to come up with a reasonable way of working with version management in DB and was inspired by the commands specified here. For my team and I, it makes no sense to put the requirements file in the dbfs locatio...
- 16016 Views
- 3 replies
- 1 kudos
- 2575 Views
- 4 replies
- 0 kudos
Unity Catalog storage amounts
Hi,I am using Azure and I do have predictive optimization enable on the catalog. I have wrote a script to calculate the data amounts of all of the tables - looping over all of the tables and running "describe detail".All of the tables amount to ~ 1....
- 2575 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @notwarte,Using the __databricks_internal catalog to trace the underlying storage location is a solid approach for investigating their footprint.Regarding your question about storage duplication: yes, materialized views in Databricks do store a p...
- 0 kudos
- 1178 Views
- 1 replies
- 0 kudos
Parametrize DLT pipeline
If I'm using Databricks Asset Bundles, how would I parametrize a DLT pipeline based on a static configuration file.In pseudo-code, I would have a .py-file:import dlt # Something that pulls a pipeline resource (or artifact) and parses from JSON table...
- 1178 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello!To parametrize a Databricks DLT pipeline with a static configuration file using Asset Bundles, include your JSON/YAML config file in the bundle. In your DLT pipeline code, read this file using Python's file I/O (referencing its deployed path). ...
- 0 kudos
- 5104 Views
- 3 replies
- 2 kudos
Resolved! DAB | Set tag based on job parameter
Hi Community.Since I wasn't able to find a way to set a job tag dynamically at runtime based on a parameter that is passed to the job, I was wondering if it is possible or if there is an equivalent way to do it.Thank you. Regards.
- 5104 Views
- 3 replies
- 2 kudos
- 2 kudos
Based on the provided context, it appears that there isn't a direct way within Databricks to dynamically set job tags at runtime based on a parameter passed to the job. However, there are alternative approaches you can consider to work around this li...
- 2 kudos
- 2568 Views
- 4 replies
- 3 kudos
How do I deploy or run one job if I have multiple jobs in a Databricks Asset Bundle?
How do I deploy or run a single job if I have 2 or more jobs defined in my asset bundle?$databricks bundle deploy job1 #does not work I do not see a flag to identify what job to run.
- 2568 Views
- 4 replies
- 3 kudos
- 3 kudos
I haven't done it with multiple jobs, but I think under resources you name multiple jobs, then when you deploy you just call that job key.
- 3 kudos
- 4879 Views
- 2 replies
- 1 kudos
[STREAMING_TABLE_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE]
Currently trying to refresh a Delta Live Table using a Full Refresh but an error keeps coming up saying that we have to use a shared cluster or a SQL warehouse. I've tried both a shared cluster and a SQL warehouse and the same error keeps coming up. ...
- 4879 Views
- 2 replies
- 1 kudos
- 1 kudos
You are not using "No Isolation Shared" mode, right? Also, can you share the chunk of code that is causing the failure? Thanks, Louis.
- 1 kudos
- 2863 Views
- 6 replies
- 3 kudos
Spark UI Simulator Not Accessible
Hello,The Spark UI Simulator is not accessible since the last few days. I was able to refer to it last week, at https://www.databricks.training/spark-ui-simulator/index.html. I already have access to partner academy (if that is any relevant). <Error...
- 2863 Views
- 6 replies
- 3 kudos
- 3 kudos
Just a short update: the request I raised was closed saying there is no active support contract with the org (from the email I used) to look into this. Perhaps someone else could try raising a request using the link above.
- 3 kudos
- 1802 Views
- 3 replies
- 1 kudos
Data Migration using Bladebridge
Hi,We are planning to migrate from Teradata to Databricks using Bladebridge. Going through various portals, I am not able to conclude the component that facilitates Data movement between Teradata and databricks.Please clarify end to end tool and acti...
- 1802 Views
- 3 replies
- 1 kudos
- 1 kudos
I'm not aware if blade bridge has data movement tool handy with them.I don't see anything advertised by them though.Let me know if you find anything on this.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 570 | |
| 349 | |
| 287 |
