- 1372 Views
- 1 replies
- 3 kudos
Why does DLT CDC some time manifests the results table as a table and other times as a view?
I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:why is my results table a view and not a table like I ...
- 1372 Views
- 1 replies
- 3 kudos
- 3 kudos
I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.
- 3 kudos
- 2799 Views
- 2 replies
- 3 kudos
Giving new user workspace access
Hey all,We have a new user we'd like to give access to our spark workspace. We invited the user to the workspace as an account admin. They click on the invite link and create a password and login. Once logged in they can see the workspace and can ...
- 2799 Views
- 2 replies
- 3 kudos
- 3 kudos
The new feature in data bricks is identity federation if identity federation is enabled then the users part of the data bricks account and the account admin can assign the users to the workspace. The account admins can add the users from account cons...
- 3 kudos
- 2168 Views
- 3 replies
- 6 kudos
Resolved! Certification Spark - Can I please get an update on when should I receive the certificate?
- 2168 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Mônica Borges Silva​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 6 kudos
- 16285 Views
- 1 replies
- 7 kudos
Resolved! How do I resolve problems when deploying a workspace with AWS Quickstart cloud formation template?
I am unable to deploy a workspace on AWS using Quickstart from my account console.Short description-You might receive one of the following common errors users face:Wrong credentialsElastic IP and VPC limit reachedRegion unavailableResolution-Wrong cr...
- 16285 Views
- 1 replies
- 7 kudos
- 7 kudos
Really great explanation. The error that I was encountering since yesterday was Failed to create CreateStorageConfiguraiton and CreateCredentialConfiguration. The first step to put the password manually helped me to solve the issue
- 7 kudos
- 1895 Views
- 3 replies
- 2 kudos
Why there's no "New Union" option with Databricks connection
I'm trying to use databricks connect with tableau but it doesn't give me the "New Union" option. Is this normal or it's particular to me? My tableau desktop version is 2021.3
- 1895 Views
- 3 replies
- 2 kudos
- 2 kudos
there is option for connecting tableau find there SPARK SQL then it should work after adding proper connection string
- 2 kudos
- 27941 Views
- 13 replies
- 13 kudos
Certified Data Engineer Associate - v2 vs. v3 (Databricks Academy)
Which version of the Data Engineering with Databricks learning plan should I do? v2 or v3? Is there a Certified Data Engineer Associate V3 Exam already?Where can I find practice exams for Certified Data Engineer Associate V3?
- 27941 Views
- 13 replies
- 13 kudos
- 13 kudos
I would suggest choose v3 - it was latest version and covered more topic.
- 13 kudos
- 3917 Views
- 4 replies
- 7 kudos
How to write a Delta Live Table(dlt) pipeline output to Databricks SQL directly
Hi,I am trying to see if it is possible to setup a direct connection from dlt pipeline to a table in Databricks SQL by configuring the Target Schema: with poc being a location of schema like "dbfs:/***/***/***/poc.db The error message was just a...
- 3917 Views
- 4 replies
- 7 kudos
- 7 kudos
When ever you store a Delta Table to Hive Metastore. This table will be available in Databricks SQL Workspace ( Data Explorer ) under hive_metastore catalog.
- 7 kudos
- 1469 Views
- 1 replies
- 3 kudos
creation of tables with cdc
I am using cdc to create different tables, these tables can have one or more dependencies, what is the best practice to create these tables without losing records or changes in both the base table and the join tables? for exampleselect * from ( ...
- 1469 Views
- 1 replies
- 3 kudos
- 3 kudos
more detail
- 3 kudos
- 2622 Views
- 1 replies
- 5 kudos
Resolved! Where can we use Broadcast variable?
best situations where we can use broadcast variables ?
- 2622 Views
- 1 replies
- 5 kudos
- 5 kudos
hey @Punit Chauhan​ BV are used in the same way for RDD, DataFrame, and Dataset.When you run a Spark RDD, DataFrame jobs that has the Broadcast variables defined and used, Spark does the following.Spark breaks the job into stages that have distribute...
- 5 kudos
- 2727 Views
- 3 replies
- 3 kudos
databricks autoloader getting stuck in flattening json files for different scenarios similar in nature.
I have a databricks autoloader notebook that reads json files from an input location and writes the flattened version of json files to an output location. However, the notebook is behaving differently for two different but similar scenarios as descri...
- 2727 Views
- 3 replies
- 3 kudos
- 3 kudos
Could you provide a code snippet? also do you see any error logs in the driver logs?
- 3 kudos
- 1139 Views
- 1 replies
- 5 kudos
PrivilegesSELECT: gives read access to an object.CREATE: gives ability to create an object (for example, a table in a schema).MODIFY: gives ability to...
PrivilegesSELECT: gives read access to an object.CREATE: gives ability to create an object (for example, a table in a schema).MODIFY: gives ability to add, delete, and modify data to or from an object.USAGE: does not give any abilities, but is an add...
- 1139 Views
- 1 replies
- 5 kudos
- 5 kudos
thanks sir
- 5 kudos
- 2157 Views
- 2 replies
- 2 kudos
Disable access to mount point for client code
We are building a platform where we automatically execute Databricks jobs using Python packages delivered by our end-users. We want to create a mount point so that we can deliver the cluster's driver logs to an external storage. However, we don't wan...
- 2157 Views
- 2 replies
- 2 kudos
- 3779 Views
- 1 replies
- 36 kudos
Understand Trigger Intervals in Streaming Pipelines in Databricks When defining a streaming write, the trigger the method specifies when the system sh...
Understand Trigger Intervals in Streaming Pipelines in DatabricksWhen defining a streaming write, the trigger the method specifies when the system should process the next set of data. Triggers are specified when defining how data will be written to a...
- 3779 Views
- 1 replies
- 36 kudos
- 36 kudos
Thank you for sharing
- 36 kudos
- 6500 Views
- 4 replies
- 3 kudos
Resolved! Can't read large multiline json,
Hey! So I'm struggling to read a multiline json. Some details:It's gzipped from the API I get it fromjust a single file in the folder currrentlystored in ADLS Gen2 storage. 95 MB zipped, approximately 1.2 GB unzippedI can read it just fine using the...
- 6500 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks a lot for the help! Removing colon fixed it. Now I need to fix the Data Factory instance that writes to my storage container. Hope it's easy, Data Factory is such a hassle.
- 3 kudos
- 2107 Views
- 3 replies
- 3 kudos
500550
[Simba][SimbaSparkJDBCDriver] (500550) The next rowset buffer is already marked as consumed. The fetch thread might have terminated unexpectedly. Foreground thread ID
- 2107 Views
- 3 replies
- 3 kudos