- 5606 Views
- 2 replies
- 1 kudos
Error in SQL statement: KeyProviderException: Failure to initialize configuration
I have a source delta table from which I have selectively granted access to a particular pool id(can be thought of a dummy user). From the pool id interface, whenever I am running a select on any of the tables, even though it has access to, is faili...
- 5606 Views
- 2 replies
- 1 kudos
- 5118 Views
- 4 replies
- 3 kudos
Resolved! Azure Databricks: Failed to extract data which is between two timestamps within those same dates using Pyspark
Data type:AAPL_Time: timestampAAPL_Close: floatRaw Data:AAPL_Time AAPL_Close 2015-05-11T08:00:00.000+0000 29.0344 2015-05-11T08:30:00.000+0000 29.0187 2015-05-11T09:00:00.000+0000 29.0346 2015-05-11T09:3...
- 5118 Views
- 4 replies
- 3 kudos
- 3 kudos
Another thing to try is the hour() and minute() functions will return integers.
- 3 kudos
- 22983 Views
- 5 replies
- 3 kudos
Resolved! Precision and scale is getting changed in the dataframe while casting to decimal
When i run the below query in databricks sql the Precision and scale of the decimal column is getting changed.Select typeof(COALESCE(Cast(3.45 as decimal(15,6)),0));o/p: decimal(16,6)expected o/p: decimal(15,6)Any reason why the Precision and scale i...
- 22983 Views
- 5 replies
- 3 kudos
- 3 kudos
You can use typeof(COALESCE(Cast(3.45 as decimal(15,6)),0.0)); (instead of 0)
- 3 kudos
- 1674 Views
- 0 replies
- 0 kudos
Easy way to debug databricks code. Is there breakpoints in databricks or alternative way to achieve it?
I'm consuming multiple topics from confluent kafka and process each row with business rules using Spark structured streaming (.writestream and .foreach()). While doing that i call other notebook using %run and call the class via foreach while perform...
- 1674 Views
- 0 replies
- 0 kudos
- 1811 Views
- 0 replies
- 3 kudos
Longer execution time to write into the SQL server table from Spark Dataframe
I have 8gb of XML data loaded into different dataframes, there are two dataframes which has 24 lakh and 82 lakh data to be written to a 2 SQL server tables which is taking so 2 hrs and 5 hrs of time to write it. I am using the below cluster configura...
- 1811 Views
- 0 replies
- 3 kudos
- 11989 Views
- 5 replies
- 7 kudos
Resolved! SQL Error when querying any tables/views on a Databricks cluster via Dbeaver.
I am able to connect to the cluster, browse its hive catalog, see tables/views and columns/datatypesRunning a simple select statement from a view on a parquet file produces this error and no other results:"SQL Error [500540] [HY000]: [Databricks][Dat...
- 11989 Views
- 5 replies
- 7 kudos
- 7 kudos
Update. I have tried SQL Workbench/J and encountered exactly the same error(s) as with Dbeaver. I have also tried JetBrains DataGrip and it worked flawlessly. Able to connect, browse the databases and query tables/views. https://docs.microsoft.com/en...
- 7 kudos
- 3781 Views
- 1 replies
- 0 kudos
Resolved! Drop/Create tables in Redshift with PySpark
Happy Friday afternoon fellow Bricksters! Got another question for you... I have a pyspark notebook that reads from redshift into a DF, does some 'stuff', then writes back to redshift. All good here. What I'm trying to do with no luck yet is first DR...
- 3781 Views
- 1 replies
- 0 kudos
- 0 kudos
Answered my own question!! check this out:dropSQL = ("DROP TABLE IF EXISTS <tablename>;"). --note the semicolon at the end!createSQL = ("CREATE TABLE IF NOT EXISTS <tablename> (field1 int, field2 date, etc...);")preActionsSQL = dropSQL + createSQLth...
- 0 kudos
- 3833 Views
- 2 replies
- 4 kudos
databricks-connect throws an exception when showing a dataframe with json content
I'm facing an issue when I want to show a dataframe with JSON content.All this happens when the script runs in databricks-connect from VS Code.Basically, I would like any help or guidance to get this run as it should be. Thanks in advance.This is how...
- 3833 Views
- 2 replies
- 4 kudos
- 4 kudos
The code works fine on databricks cluster, but this code is part of a unit test in local env. then submitted to a branch->PR->merged into master branch.Thanks for the advice on using DBX. I will give DBX a try again even though I've already tried.I'l...
- 4 kudos
- 1121 Views
- 1 replies
- 0 kudos
Hi,I'll like to know if it's possible to connect to Postgresql RDS from the Databricks SQL Warehouse.
Hi,I'll like to know if it's possible to connect to Postgresql RDS from the Databricks SQL Warehouse.
- 1121 Views
- 1 replies
- 0 kudos
- 0 kudos
I should have posted this as a question and not a post. Please forgive me, I'm a newbie.
- 0 kudos
- 2025 Views
- 3 replies
- 1 kudos
What's the be​at way to define the libraries for cluster as it always take a lot of time for me.
What's the be​at way to define the libraries for cluster as it always take a lot of time for me.
- 2025 Views
- 3 replies
- 1 kudos
- 1 kudos
@Nikunj Goel​ : Please refer to the below doc the workspace library might help on thishttps://docs.databricks.com/libraries/workspace-libraries.html#workspace-libraries
- 1 kudos
- 1505 Views
- 0 replies
- 0 kudos
i tried to pull the report from QuickBase but it is giving error report too large
hii tried to pull the report from below query%pythondf = quickbasePull('b5zj8k_pbz5_0_cd5h4wbbp77n4nvp56b4u','bqmnP8jm7',24)but it is giving me error report too largethen i tried below%pythonimport pyqbfrom pyspark.sql import *import pandas as pdqbc ...
- 1505 Views
- 0 replies
- 0 kudos
- 2616 Views
- 0 replies
- 2 kudos
Use output of SHOW PARTITION commands in Sub-Query/CTE/Function
I am using SHOW PARTITIONS <<table_name>> to get all the partitions of a table. I want to use max() on the output of this command to get the latest partition for the table.However, I am not able to use SHOW PARTITIONS <<table_name>> in a CTE/sub-quer...
- 2616 Views
- 0 replies
- 2 kudos
- 868 Views
- 0 replies
- 2 kudos
Want to influence the Databricks product roadmap and services? Â We are looking for feedback from you - our Databricks Community members - to give your...
Want to influence the Databricks product roadmap and services? We are looking for feedback from you - our Databricks Community members - to give your feedback and thoughts about your experience with Databricks over the last 6 months in a ~10 minute s...
- 868 Views
- 0 replies
- 2 kudos
- 1819 Views
- 0 replies
- 1 kudos
Delta Live Tables executing repeatedly and returning empty DF
Still relatively new to Spark and even more so to Delta Live Tables so apologies if I've missed something fundamental but here goes.We are trying to run a notebook via Delta Live Tables, which contains 2 functions decorated by the `dlt.table` decorat...
- 1819 Views
- 0 replies
- 1 kudos
- 4234 Views
- 1 replies
- 0 kudos
Applying a formula to list of python dataframes produces error: object of type 'builtin_function_or_method' has no len(). How to fix?
I have a df where I am calculating values by month. When I run this code on my df it generates the desired results:for i in range(12,len(df.index)): df.iloc[i, 1] = df.iloc[i-12,1]*(((df.iloc[i,3]/100)+(df.iloc[i,6]/100))+1)So far so good. I want...
- 4234 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey there @Jack Homareau​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
