- 2090 Views
- 6 replies
- 0 kudos
Resolved! Retrieve error column/row when writing to sqlmi
Databricks notebook, Scala, .read() .write(), source data csv. Got this error when trying to write data to sqlmi. I understand the error indicated one of the source column value's length exceeded the database column's value length. But the message is...
- 2090 Views
- 6 replies
- 0 kudos
- 0 kudos
Thanks JamesY, I'm not familiar with the limitations of the SQL Server `nvarchar` data type but is there a way that we can filter out the rows that will fail using spark such as: spark.read.format("csv").option("header", "true").load("/path/to/csvs")...
- 0 kudos
- 2376 Views
- 5 replies
- 1 kudos
Resolved! [TABLE_OR_VIEW_ALREADY_EXISTS] when running create or replace view
When I'm running `dbt run -s model` for different models in parallel on databricks general compute cluster. Im getting the error: TABLE_OR_VIEW_ALREADY_EXISTS. Also, at the same time, the view/table was not created at all.But I run them in sequence, ...
- 2376 Views
- 5 replies
- 1 kudos
- 1 kudos
@raphaelblg yes! Im the owner. However, these views are inside the hive_metastore catalog. Could it be the reason?Many thaaanks!
- 1 kudos
- 2941 Views
- 2 replies
- 0 kudos
InconsistentReadException: The file might have been updated during query - CSV backed table
I have some CSV files that I upload to DBFS storage several times a day. From these CSVs, I have created SQL tables: CREATE TABLE IF NOT EXISTS masterdata.lookup_host USING CSV OPTIONS (header "true", inferSchema "true") LOCATION '/mnt/masterdata/...
- 2941 Views
- 2 replies
- 0 kudos
- 0 kudos
One approach I'm testing (positive results so far, but still early).%sql # Prep and cleanup REFRESH TABLE masterdata.lookup_host; DROP TABLE IF EXISTS t_hosts; # Forcibly cache the needed columns before using the data in another query. CACHE TABLE...
- 0 kudos
- 12841 Views
- 0 replies
- 0 kudos
Change adls gen2 attached to Unity Catalog metastore from premium to standard
Hello,Our cloud platform engineer created Azure storage for the production Unity Catalog metastore in our environment, but mistakenly chose the Premium tier instead of the Standard tier.Unfortunately, this decision is impacting our costs on Azure, as...
- 12841 Views
- 0 replies
- 0 kudos
- 1709 Views
- 2 replies
- 0 kudos
Accessing ADLS Gen2 related Hadoop configuration in notebook
I have a cluster in which I have the required configuration to access an ADLS Gen2, and it works without any problems.I want to access this storage using the Hadoop filesystem APIs. To achieve this, I am trying to get the Hadoop configuration from th...
- 1709 Views
- 2 replies
- 0 kudos
- 0 kudos
By the way how do you achieve inline code highlighting in the editor I tried `` but it didn't work.
- 0 kudos
- 1062 Views
- 1 replies
- 2 kudos
Resolved! DAIS24 Attendee Badge - Oops
I wasn't at DAIS24, but I received one of these emails and it appears to have come from Databricks. I think I was given the badge in error. Can we have it removed?Just Wondering,Shawn
- 1062 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Shawn_Eary Thank you for bringing this to our attention. We were conducting a test on badges, and some of our community members may have received this email by accident. We have revoked the changes now, please ignore the message.
- 2 kudos
- 5933 Views
- 1 replies
- 0 kudos
Cluster Memory Issue (Termination)
Hi,I have a single-node personal cluster with 56GB memory(Node type: Standard_DS5_v2, runtime: 14.3 LTS ML). The same configuration is done for the job cluster as well and the following problem applies to both clusters:To start with: once I start my ...
- 5933 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I have a single-node personal cluster with 56GB memory(Node type: Standard_DS5_v2, runtime: 14.3 LTS ML). The same configuration is done for the job cluster as well and the following problem applies to both clusters:To start with: once I start my ...
- 0 kudos
- 1406 Views
- 1 replies
- 0 kudos
Tableau Prep Save Output to Databricks
Has anyone run into use cases where your data scientist/data engineer end users build Tableau Prep Flows and steps in Tableau Prep Flow require saving output back into Databricks? There appears to be no native support for this in Tableau Prep if the ...
- 1406 Views
- 1 replies
- 0 kudos
- 0 kudos
These are awesome suggestions. To expand on our setup, we also have Informatica Cloud - IICS (CMI, CDI, etc.) connected to the entire setup generally used for bringing data from a source (PaaS, SaaS, On-prem SQL, Flat Files or streaming devices) to D...
- 0 kudos
- 655 Views
- 0 replies
- 0 kudos
Training link deprecated: How to ingest data for Databricks SQL
I am currently doing a course in Databricks academy: How to ingest data for Databricks SQL.To create a table in the external location I am provided with the link that is not working anymore. Below is the link:wasbs://courseware@dbacademy.blob.core.wi...
- 655 Views
- 0 replies
- 0 kudos
- 7345 Views
- 2 replies
- 0 kudos
Seeking Tips: Ways to Master Databricks on Azure?
Hello everyone. I'm currently learning Databricks on Azure through a Udemy course. Recently, I was surprised by a charge of $86 from Azure, which has made me cautious about continuing in this manner. Is there a more cost-effective approach to learn D...
- 7345 Views
- 2 replies
- 0 kudos
- 570 Views
- 1 replies
- 0 kudos
Assigning a group as USER to service principal
How can we assigning a group as USER to service principal using databricks-sdk, this is not supported?
- 570 Views
- 1 replies
- 0 kudos
- 0 kudos
I found this API, https://docs.databricks.com/api/account/accountaccesscontrol/updaterulesetbut its PUT and GET method both requires a parameter "etag", how can someone know this "etag"?
- 0 kudos
- 3749 Views
- 6 replies
- 2 kudos
Unity catalog enabled workspace -Is there any way to disable workflow/job creation for certain users
Currently in unity catalog enabled workspace users with "Workspace access" can create workflows/jobs, there is no access control available to restrict users from creating jobs/workflows.Use case: In production there is no need for users, data enginee...
- 3749 Views
- 6 replies
- 2 kudos
- 2 kudos
not being able to restrict creation of workflows/jobs, alerts, dashboards make platform admin job difficult to keep system clean and control cost. there is no need for data engineers to create a workflow in production. there is no need for all users ...
- 2 kudos
- 1505 Views
- 3 replies
- 0 kudos
List granted access for a group or a user
Is there any way where I can see what access a group or a user have been given to objects (Tables, views, catalogs etc.)?I noticed that we have the following information_schema tables:catalog_privilegesroutine_privilegestable_privilegesschema_privile...
- 1505 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Henrik To grant a user the privilege to query system tables, a metastore admin or another privileged user must grant USE and SELECT permissions on the system schemas. GRANT USAGE ON CATALOG system TO <user_name>; GRANT USAGE ON SCHEMA informa...
- 0 kudos
- 957 Views
- 0 replies
- 0 kudos
Databricks + confluent schema registry - Schema not found error
I am running a Kafka producer code on Databricks 12.2. I am testing AVRO serialization of message with help of confluent schema registry. I configured 'to_avro' function to read the schema from schema registry, but I am getting the below error> org.a...
- 957 Views
- 0 replies
- 0 kudos
- 930 Views
- 1 replies
- 0 kudos
Materialized view in DLT pipeline
When setting up DLT pipeline, there are 3 types of product edition, Core, Pro and Advanced. When I compare DLT Classic Core and DLT Classic Pro, the difference is that DLT Classic Pro can handle CDC. Does it means if I'm using DLT Classic Core, I hav...
- 930 Views
- 1 replies
- 0 kudos
- 0 kudos
Also, If I'm using DLT Classic Core for DLT pipeline, the materialized view will be doing a full refresh or just update on the rows that have changes?
- 0 kudos
