- 3477 Views
- 2 replies
- 0 kudos
Autoloader, toTable
"In autoloader there is the option ".toTable(catalog.volume.table_name)", I have an autoloder script that reads all the files from a source volume in unity catalog, inside the source I have two different files with two different schemas.I want to sen...
- 3477 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @ShlomoSQM, looks like @shan_chandra suggested a feasible solution, just to add a little more context this is how you can achieve the same if you have a column that can help you identify what is type1 and type 2file_type1_stream = readStream.opti...
- 0 kudos
- 5324 Views
- 3 replies
- 0 kudos
Global or environment parameters.
Hi All,Need a help on creating utility file that can be use in pyspark notebook.Utility file contain variables like database and schema names. So I need to pass this variables in other notebook wherever I am using database and schema.Thanks
- 5324 Views
- 3 replies
- 0 kudos
- 0 kudos
You can use:${param_catalog}.schema.tablename.Pass actual value in the notebook through a job param "param_catalog" or widget utils through text called "param_catalog"
- 0 kudos
- 3969 Views
- 1 replies
- 0 kudos
Schema inference with auto loader (non-DLT and DLT)
Hi. Another question, this time about schema inference and column types. I have dabbled with DLT and structured streaming with auto loader (as in, not DLT). My data source use case is json files, which contain nested structures. I noticed that in t...
- 3969 Views
- 1 replies
- 0 kudos
- 2845 Views
- 3 replies
- 0 kudos
Unable to DROP TABLE: "Lock wait timeout exceeded"
We have a table where the underlying data has been dropped, and seemingly something else must have gone wrong as well, and we want to just get rid of the whole table and schema, but running "DROP TABLE schema.table" is throwing the following error:or...
- 2845 Views
- 3 replies
- 0 kudos
- 0 kudos
The table needs to be dropped from the backend. If you can raise a ticket, support team can do it for you.
- 0 kudos
- 9657 Views
- 5 replies
- 0 kudos
Resolved! Need to define the struct and array of struct field colum in the delta live table(dlt) in databrick.
I want to create the columns with datatype struct and array of struct datatype in the DLT live tables, will it be possible, if possible could you share the sample for the same.Thanks.
- 9657 Views
- 5 replies
- 0 kudos
- 0 kudos
I have created DLT live tables pipeline, In Job UI, i can able to see only steps and if any failure happened it show only error at that stage.But if i use any log using print, it doesn't show the logs in the console or any where. how can i see the lo...
- 0 kudos
- 4479 Views
- 3 replies
- 1 kudos
Resolved! IsBlindAppend config changes
Hello Allcan someone please suggest me how can I change the config IsBlindAppend true from false. I need to do this not for a data table but a custom log table .Also is there any concern If I toggle the value as standard practices. pls suggest
- 4479 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, IsBlindAppend is not a config but an operation metrics that is used in Delta Lake History. The value of this changes based on the type of operation performed on Delta table. https://docs.databricks.com/en/delta/history.html
- 1 kudos
- 5978 Views
- 5 replies
- 3 kudos
Resolved! terraform create multiple db user
Hi, follow the example to create one user. It's working however I want to create multiple users, I have tried many ways but still cannot get it work, please share some idea.https://registry.terraform.io/providers/databricks/databricks/latest/docs/res...
- 5978 Views
- 5 replies
- 3 kudos
- 3 kudos
What if I want to give User Name along with the email ID?I used below code but its not helping(code is not failing, but not adding user name)It seems this code line: "display_name = each.key" is not working. Pls suggest. terraform {required_provider...
- 3 kudos
- 2889 Views
- 2 replies
- 0 kudos
java.io.IOException: Invalid PKCS8 data error when reading data from Google Storage
Databricks workspace is hosted in AWS. Trying to access data in Google Cloud Platform.I have followed the instructions here: https://docs.databricks.com/en/connect/storage/gcs.htmlI get error: "java.io.IOException: Invalid PKCS8 data." when trying t...
- 2889 Views
- 2 replies
- 0 kudos
- 13572 Views
- 1 replies
- 0 kudos
DLT quarantine records
How to capture bad records that are violating expectations into quarantine tables, can someone provide DLT SQL code syntax for the same
- 13572 Views
- 1 replies
- 0 kudos
- 0 kudos
I would like to share the following docs, which will have examples https://docs.databricks.com/en/delta-live-tables/expectations.html
- 0 kudos
- 2152 Views
- 1 replies
- 0 kudos
Error while performing async I/O for file
We're running dbt Cloud on DBSQL. And a frequent error we keep getting in our dbt jobs is "Error while performing async I/O for file [S3 URI path]". Since we don't have access to the full logs it's very difficult to know what's actually going on her...
- 2152 Views
- 1 replies
- 0 kudos
- 0 kudos
do you have access to create a support ticket? if you do, we can retrieve the logs for you and provide the details. If you dont, then you will need access to your driver's logs to identify the root cause of this issue.
- 0 kudos
- 2644 Views
- 2 replies
- 1 kudos
How to writeStream with redshift
I have already checked the documentation below The documentation below does not describe how to write to streaming.Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in...
- 2644 Views
- 2 replies
- 1 kudos
- 2023 Views
- 1 replies
- 0 kudos
Module not found when using applyInPandasWithState in Repos
I should start by saying that everything works fine if I copy and paste it all into a notebook and run it. The problem starts if we try to have any structure in our application repository. Also, so far we have only run into this problem with applyInP...
- 2023 Views
- 1 replies
- 0 kudos
- 0 kudos
which DBR version are you using? does it works on non DLT jobs?
- 0 kudos
- 1651 Views
- 1 replies
- 0 kudos
did anyone faced this issue in delta table while genrating manifest file
error message : Manifest generation is not supported for tables that leverage column mapping, as external readers cannot read these Delta tableswhy i got this issue. not sure should we need to do any process ?
- 1651 Views
- 1 replies
- 0 kudos
- 0 kudos
could you please share the full stack trace and the repro steps?
- 0 kudos
- 1499 Views
- 1 replies
- 0 kudos
Not able to load nested XML file with struct type
Hello Experts,I am trying to load XML with struct type and having XSI type attribute. below is sample XML format:<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="htt...
- 1499 Views
- 1 replies
- 0 kudos
- 0 kudos
You can try to use from_xml() function, here is the link to the docs https://docs.databricks.com/en/sql/language-manual/functions/from_xml.html
- 0 kudos
- 2363 Views
- 1 replies
- 1 kudos
Databricks Deployment using Data Thirst
Hi,I am trying to deploy Databricks Notebooks using Azure Devops to different environments using third party extension Data Thirst (Databricks Script Deployment Task by Data Thirst). The pipeline is able to generate/download artifacts but not able to...
- 2363 Views
- 1 replies
- 1 kudos
- 1 kudos
the extension is quite old and does not know about Unity Catalog. So that is probably the reason why it fails.But why do you use the extension for notebook propagation from dev to prd? You can do this using Repos, feature branches and pull requests...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
