Hi team,I was giving my exam today and 40 minutes into the exam I was interrupted by the proctor to show the test area. The table had a guitar e string and an almost eaten apple. Nothing else was on the table. Then the proctor asked me to show the ro...
I'm trying to follow along with the blog post Gaining Insights Into Your HL7 Data With Smolder and Databricks-#1 of 3. I was able to finally get a jar file built from the repo using Java 17 and it successfully imports into the cluster. However, when ...
I need to update a SQL Server Table from Databricks notebook. Right now, I am trying to do this using JDBC. However, it seems we can only append or overwrite the table using the JDBC Connection.Query databases using JDBC - Azure Databricks | Microsof...
Hi @berserkersap thanks for your answer. I was able to solve the problem in 2 ways:1) downgrading the Runtime version to 12.2 and then the installer of msodbcsql17 no longer failed (with the error Can't open lib 'ODBC Driver 17 for SQL Server' : file...
Hello everybody,i tried to build a Logic App Custom Connector following this one explanation. (https://medium.com/@poojaanilshinde/create-azure-logic-apps-custom-connector-for-azure-databricks-e51f4524ab27)Now i run in the following Problem and wante...
Hey Lukas,I can totally relate to the frustration of encountering those confounding errors when building custom connectors in Azure Logic Apps. The "Job ID value out of range" issue can be quite perplexing, but fear not, for there's a solution on the...
I am trying to query my postgresql read replica as a foreign catalog.I can sucessfuly test the connection.I can see the database names.The table names are auto populated correctly.However when I try to view or query a table, I get the following error...
Hi @Retired_mod I am able to query the pg_catalog database which has all lower case table names, so the connection is working.I am unable to query the tables in my "public" schema, as they have capitalization in the table names.If I query with no bac...
Hello everyone,I have a .py file (not a notebook) where I have the following class with the following constructor:class DataQualityChecker: def __init__(self, spark_session: SparkSession, df: DataFrame, quality_config_filepath: str) -> None: ...
Hello everyone,I get a weird error when installing additional libraries in my cluster.I have a predefined Databricks cluster (Standard_L8s_v2) as a Compute instance. I run pipelines on that cluster in Azure ADF. The pipeline consists several tasks. T...
As a DevSecOps engineer, I want to provide Ops support personnel READ ONLY access to the admin console in my production workspaces, so that they can easily view non-secret configurations, such as user/group memberships/entitlements and workspace sett...
Hello,I have an error on a databricks workspace (production) when trying to import a local module into azure databricks notebook. I work with a Databricks Repos.I have the message : "ModuleNotFoundError: No module named 'test.dataset_test'"When I tes...
See attached screenshot. In my SQL notebook, declare a temporary view:CREATE OR REPLACE TEMP VIEW tmp_table ASSELECT ...;SELECT count(*) FROM tmp_table; The code editor considers tmp_table is not a valid name in that latter SELECT. The reason is:Coul...
My temp views always show red as well. Maybe it is something with our specific environment?I've also noticed, when browsing the catalog from within the notebook, the temp tables are listed but an error is thrown when you try to view the columns of t...
Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...
public void bigDataTest() throws Exception { int rowsCount = 100_000; int colSize = 1024; int colCount = 12; String colValue = "'"+"x".repeat(colSize)+"'"; String query = "select explode(s...
I am genuinely curious why would you ever not use Autoloader? I see it in one-off downloads of course. When you pull data from another platform, say Salesforce, is it better to append to a table without Autoloader? There must be cases I am missing. T...
Autoloader is pretty handy, but not open source. That is one reason f.e.Another reason is f.e. if you cannot guarantee lexicographically generated files, or you do not want to use streaming, or you do not land your raw data into a data lake (read fr...
Hi All,, quick question:Is this correct data flow pattern: Databricks -> Az SQL -> Tableau??Or does it have to go through ADLS: Databricks -> ADLS -> Az SQL - > Tableau? Also, is it better to leverage databricks lakehouse sql warehouse capability as ...
I would not call it 'better' per se. A lakehouse is a more modern approach to a classic datawarehouse, using flexible distributed cloud compute, cheap storage and open file formats.If you have an existing environment, which works well, that is heavi...
Hi, I'm getting this error with Power BI:Any ideas?Thanks in advance,Andres
Hi Harrison, thanks for the reply. We have sorted the issue a few days later but forgot to reply to your answer. There had been some changes on the security set-up that affected our connectivity to Databricks from all clients.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group
